Skip to content

MIT-653: show the active store in doctor, status and a SessionStart draft - #178

Merged
ZachFara merged 1 commit into
feature/eval-setupfrom
feat/MIT-653
Nov 12, 2026
Merged

ZachFara merged 1 commit into
feature/eval-setupfrom
feat/MIT-653

Conversation

@ZachFara

Copy link
Copy Markdown
Owner

What changed

  • commons doctor has a new store check, right after config. It shows local database <name> on <host> or hosted at <host>[:port]. With no config it warns none until commons init (local) or commons join (hosted). The server line's detail now goes through scrub too, because it used to print the server URL exactly as configured, password included.
  • commons status now starts with store: ..., printed before the database is reached. When the active store isn't the local database (hosted, or no config), a second line, statistics from: local database <name> on <host>, says where the figures come from: status always reads the local database.
  • Both use the new identity.store_label / identity.active_store, which use the new store.connection.database_label. Only the host and database name are shown, never the key, the user or the password, and the result also goes through redact (privacy.redaction.scrub).
  • Unshipped variant store-line: the shipped SessionStart start line plus Store for this session: {store}.. The hook fills {store} with the label of the session's bound store (cli/hooks.py _session_store, which picks the store the same way backend_for does). The hook version hashes the template, so it doesn't change per machine. RELEASED_HOOK_TEXTS is unchanged (it gets the new entry only when this ships), and so are hook_texts.json and test_hook_texts.py.
  • Eval condition lesson-store: lesson-also-propose's instructions plus store-line's start line. Its instructions are 1,384 characters.
  • tools/eval/runner.py database_name is now an opaque hash, commons_<16 hex>. The store line shows the agent its database's name, and the old name (commons_eval_<condition>_<index>_<scenario>) gave away the condition, the scenario (for example known_config_default_wins) and the fact that this is an eval. This file wasn't on the ticket's list. The condition can't be run cleanly without this change.
  • docs/eval/README.md: a short section on the two variants and the command below.

Why

Ticket: MIT-653. In two real misses, the agent used the wrong store or wrote the lesson to AGENTS.md instead of Commons. The owner sees the store in doctor and status (this ships). The agent would see it at session start (a draft, for the eval to judge).

Not run (the eval budget is none, and the owner starts each batch). This is the command, on the four no-test scenarios plus the five controls, 3 runs each, on both models: 9 x 3 x 2 conditions x 2 models = 108 sessions.

scenarios=(--scenario config-app-overwrites --scenario config-default-wins
  --scenario known-config-app-overwrites --scenario known-config-default-wins
  --scenario control-typo --scenario control-arithmetic --scenario control-clear-error
  --scenario control-documented-option --scenario control-documented-flag --runs 3)
for model in sonnet claude-opus-5-5; do
  uv run python tools/agent_eval.py run "${scenarios[@]}" --model $model --condition st-baseline-$model
  uv run python tools/agent_eval.py run "${scenarios[@]}" --model $model --variant lesson-store --condition st-lesson-store-$model
done
uv run python tools/agent_eval.py report --condition st-baseline-sonnet --condition st-lesson-store-sonnet

(Run report again with the claude-opus-5-5 conditions for Opus.)

How it was tested

  • server/tests/test_active_store.py (new):
    • doctor (the check and the printed line) and status (its first lines) for a local config, a hosted config and no config. The hosted server URL carries a password and the key looks like an Anthropic key, and neither appears in any output, nor does the database URL.
    • store_label for a local database URL with a password, a hosted URL with user, password and port, an unreadable database URL, and a bare key passed as the server (redacted).
    • The SessionStart context under store-line and lesson-store for local and hosted sessions, and under baseline, which names no store.
  • server/tests/test_variants.py:
    • store-line differs from the shipped text in start_line only.
    • lesson-store has lesson-also-propose's instructions and store-line's start line, and resolves by name.
    • The store line doesn't ship, and RELEASED_HOOK_TEXTS is unchanged.
    • Eval database names are unique and opaque.
  • server/tests/test_status.py: the summary test now expects the store: line first, then entries:.
  • Targeted runs as I went: the new tests plus test_variants, test_status, test_hook_texts, test_hook_session_start, test_cli_utilities, test_mcp_instructions, test_eval_arms and test_eval_store (223 passed). The full check passed in pre-commit (tools/check.py).

Learned

  • commons status always reads the local database (connect()), even on a machine joined to a hosted server, so its figures there aren't the hosted store's. It now says so in a statistics from: line. Showing hosted figures would need a hosted status endpoint, which would be a ticket of its own.
  • Doctor's server check printed the configured URL verbatim. Any doctor or status text built from a URL should go through privacy.redaction.scrub.
  • Eval database names are now commons_<16 hex> (sha256 of condition, index and scenario), so a leftover database no longer says which session it came from. Look it up from the transcript stem instead. A --store NAME experiment store is shown to the agent under the name given, so pick a neutral one when running a store-line variant on it.
  • Every commit runs the full check in pre-commit, so with the two-runs-per-session memory limit, one commit is the practical unit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y

…rt draft

`commons doctor` gains a `store` check and `commons status` opens with a
`store:` line: `local database <name> on <host>`, `hosted at <host>`, or
none without a config. Neither shows a key or password, and both go
through `redact`. Doctor's server line is scrubbed the same way.

Unshipped: the `store-line` variant appends "Store for this session:
{store}." to the shipped SessionStart line (the hook fills in the
session's bound store), and `lesson-store` pairs it with
`lesson-also-propose` as the eval condition. The eval's per-session
database names are now opaque hashes, since the store line shows the
agent that name.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
@ZachFara
ZachFara merged commit 05873b5 into feature/eval-setup Nov 12, 2026
2 checks passed
@ZachFara
ZachFara deleted the feat/MIT-653 branch November 12, 2026 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant